Holographic waveguide eye tracker

ABSTRACT

An eye tracker having a first waveguide for propagating illumination light along a first waveguide path and propagating image light reflected from at least one surface of an eye along a second waveguide path. At least one grating lamina for deflecting the illumination light out of the first waveguide path towards the eye and deflecting the image light into the second waveguide path towards a detector is disposed adjacent an optical surface of the waveguide.

CROSS-REFERENCE TO RELATED APPLICATIONS

This present application is a continuation of U.S. patent applicationSer. No. 16/922,858, entitled “Holographic Waveguide Eye Tracker” toPopovich et al., filed Jul. 7, 2020, which is a continuation of U.S.patent application Ser. No. 16/277,390, entitled “Holographic WaveguideEye Tracker” to Popovich et al., filed Feb. 15, 2019 and issued on Jul.14, 2020 as U.S. Pat. No. 10,712,571, which is a continuation of U.S.patent application Ser. No. 14/903,249, entitled “Holographic WaveguideEye Tracker” to Popovich et al., filed Jan. 6, 2016 and issued on Feb.19, 2019 as U.S. Pat. No. 10,209,517, which is a U.S. National Phase ofPCT Application No. PCT/GB2014/000197, entitled “Holographic WaveguideEye Tracker” to Popovich et al, filed May 19, 2014, which claims thebenefit of U.S. Provisional Patent Application No. 61/855,625, entitled“Apparatus for Eye Tracking” to Popovich et al., filed May 20, 2013, thedisclosures of which are incorporated herein by reference in theirentireties.

BACKGROUND

This invention relates to an eye tracking sensor, and more particularlyto an eye tracker using electrically switchable gratings.

Eye tracking is important in Head Mounted Displays (HMDs) because it canextend the ability of the user to designate targets well beyond the headmobility limits. Eye tracking technology based on projecting IR lightinto the users eye and utilizing the primary Purkinje reflections andthe pupil-masked retina reflection have been around since the 1980's.The method tracks the relative motion of these images in order toestablish a vector characterizing the point of regard. Most eye trackershave employed flat beam splitters in front of the users' eyes andrelatively large optics to image the reflections onto a sensor(generally a CCD or CMOS camera).

There is much prior art in the patent and scientific literatureincluding the following United States filings:

-   1. United Stated Patent Application Publication No. US2011019874    (A1) by Levola et al entitled DEVICE AND METHOD FOR DETERMINING GAZE    DIRECTION;-   2. U.S. Pat. No. 5,410,376 by Cornsweet entitled Eye tracking method    and apparatus;-   3. U.S. Pat. No. 3,804,496 by Crane et al entitled TWO DIMENSIONAL    EYE TRACKER AND METHOD FOR TRACKING AN EYE TWO DIMENSIONAL EYE    TRACKER AND METHOD FOR TRACKING AN EYE;-   4. U.S. Pat. No. 4,852,988 by Velez et al entitled Visor and camera    providing a parallax-free field-of-view image for a head-mounted eye    movement measurement system;-   5. U.S. Pat. No. 7,542,210 by Chirieleison entitled EYE TRACKING    HEAD MOUNTED DISPLAY;-   6. United Stated Patent Application Publication No. US 2002/0167462    A1 by Lewis entitled PERSONAL DISPLAY WITH VISION TRACKING; and-   7. U.S. Pat. No. 4,028,725 by Lewis entitled HIGH RESOLUTION VISION    SYSTEM.

The exit pupil of these trackers is generally limited by either the sizeof the beamsplitter or the first lens of the imaging optics. In order tomaximize the exit pupil, the imaging optics are positioned close to thebeamsplitter, and represent a vision obscuration and a safety hazard.Another known limitation with eye trackers is the field of view, whichis generally limited by the illumination scheme in combination with thegeometry of the reflected images off the cornea. The cornea is anaspheric shape of smaller radius that the eye-ball. The corneareflection tracks fairly well with angular motion until the reflectedimage falls off the edge of the cornea and onto the sclera. The need forbeam splitters and refractive lenses in conventional eye trackersresults in a bulky component that is difficult to integrate into a(HMD). The present invention addresses the need for a slim, wide fieldof view, large exit pupil, high-transparency eye tracker for HMDs.

The inventors have found that diffractive optical elements offer a routeto providing compact, transparent, wide field of view eye trackers. Oneimportant class of diffractive optical elements is based on SwitchableBragg Gratings (SBGs). SBGs are fabricated by first placing a thin filmof a mixture of photopolymerizable monomers and liquid crystal materialbetween parallel glass plates. One or both glass plates supportelectrodes, typically transparent indium tin oxide films, for applyingan electric field across the film. A volume phase grating is thenrecorded by illuminating the liquid material (often referred to as thesyrup) with two mutually coherent laser beams, which interfere to form aslanted fringe grating structure. During the recording process, themonomers polymerize and the mixture undergoes a phase separation,creating regions densely populated by liquid crystal micro-droplets,interspersed with regions of clear polymer. The alternating liquidcrystal-rich and liquid crystal-depleted regions form the fringe planesof the grating. The resulting volume phase grating can exhibit very highdiffraction efficiency, which may be controlled by the magnitude of theelectric field applied across the film. When an electric field isapplied to the grating via transparent electrodes, the naturalorientation of the LC droplets is changed causing the refractive indexmodulation of the fringes to reduce and the hologram diffractionefficiency to drop to very low levels. Note that the diffractionefficiency of the device can be adjusted, by means of the appliedvoltage, over a continuous range. The device exhibits near 100%efficiency with no voltage applied and essentially zero efficiency witha sufficiently high voltage applied. In certain types of HPDLC devicesmagnetic fields may be used to control the LC orientation. In certaintypes of HPDLC phase separation of the LC material from the polymer maybe accomplished to such a degree that no discernible droplet structureresults.

SBGs may be used to provide transmission or reflection gratings for freespace applications. SBGs may be implemented as waveguide devices inwhich the HPDLC forms either the waveguide core or an evanescentlycoupled layer in proximity to the waveguide. In one particularconfiguration to be referred to here as Substrate Guided Optics (SGO)the parallel glass plates used to form the HPDLC cell provide a totalinternal reflection (TIR) light guiding structure. Light is “coupled”out of the SBG when the switchable grating diffracts the light at anangle beyond the TIR condition. SGOs are currently of interest in arange of display and sensor applications. Although much of the earlierwork on HPDLC has been directed at reflection holograms transmissiondevices are proving to be much more versatile as optical system buildingblocks. Typically, the HPDLC used in SBGs comprise liquid crystal (LC),monomers, photoinitiator dyes, and coinitiators. The mixture frequentlyincludes a surfactant. The patent and scientific literature containsmany examples of material systems and processes that may be used tofabricate SBGs. Two fundamental patents are: U.S. Pat. No. 5,942,157 bySutherland, and U.S. Pat. No. 5,751,452 by Tanaka et al. Both filingsdescribe monomer and liquid crystal material combinations suitable forfabricating SBG devices.

One of the known attributes of transmission SBGs is that the LCmolecules tend to align normal to the grating fringe planes. The effectof the LC molecule alignment is that transmission SBGs efficientlydiffract P polarized light (ie light with the polarization vector in theplane of incidence) but have nearly zero diffraction efficiency for Spolarized light (ie light with the polarization vector normal to theplane of incidence. Transmission SBGs may not be used at near-grazingincidence as the diffraction efficiency of any grating for Ppolarization falls to zero when the included angle between the incidentand reflected light is small.

There is a requirement for a compact, lightweight eye tracker with alarge field of view, and a high degree of transparency to externallight.

SUMMARY

It is a first object of the invention to provide a compact, lightweighteye tracker with a large field of view, and a high degree oftransparency to external light.

It is a second object of the invention to provide a compact, lightweighteye tracker with a large field of view, and a high degree oftransparency to external light implemented in a thin optical waveguide.

The objects of the invention are achieved in one embodiment of theinvention in which there is provided an eye tracker comprising: a firstwaveguide for propagating illumination light along a first waveguidepath and propagating image light reflected from at least one surface ofan eye along a second waveguide path; a source of the illumination lightoptically coupled to the waveguide and a detector optically coupled tothe waveguide. At least one grating lamina for deflecting theillumination light out of the first waveguide path towards the eye anddeflecting the image light into the second waveguide path towards thedetector is disposed adjacent an optical surface of the waveguide. Theoptical surface of the waveguide is at least one of an internal surfaceor an external surface of the waveguide.

In one embodiment the grating lamina comprises an output grating fordeflecting the illumination light out of the first waveguide pathtowards eye and an input grating for deflecting the image light into thesecond waveguide path towards the detector.

In one embodiment at least one of the input and output gratingscomprises at least one switchable grating element having a diffractingstate and a non diffracting state.

In one embodiment the grating lamina comprises at least one switchablegrating element having a diffracting state and a non diffracting state.An element in the diffractin g state deflects illumination light out ofthe first waveguide path towards the eye and deflects the image lightinto the second waveguide path towards the detector.

In one embodiment the switchable grating elements are elongate withlonger dimension aligned perpendicular to at least one of the first andsecond waveguide paths.

In one embodiment the first and second waveguide paths are parallel.

In one embodiment the grating lamina further comprises at least one of:an input grating for deflecting illumination light from the source intothe first waveguide path; and an output grating for deflecting imagelight out of the second waveguide path towards the detector.

In one embodiment the grating lamina further comprises a secondwaveguide containing a linear array of switchable grating elementsoptically coupled to the detector and overlaying the output grating.Each element when in its diffracting state samples a portion of thelight in the first waveguide and deflects it along the second waveguidetowards the detector.

In one embodiment the grating lamina further comprises a third waveguidecontaining a linear array of switchable grating elements opticallycoupled to the light source and overlaying the input grating. Eachelement when in its diffracting state deflects light from the thirdwaveguide into the first waveguide.

In one embodiment the output grating abuts an upper or lower edge of theoutput grating along the first waveguide path.

In one embodiment the output grating comprises upper and lower gratingsdisposed adjacent upper and lower edges of the output grating along thefirst waveguide path.

In one embodiment the input grating comprises a first array ofswitchable elongate beam deflection grating elements and an overlappingsecond array of switchable elongate beam deflection grating elements.The elements of the first and second arrays are disposed with theirlonger dimensions orthogonal.

In one embodiment at least one of the input and output gratings is alinear array of elongate switchable beam deflection elements with longerdimension aligned perpendicular to the first and second waveguide paths.

In one embodiment the grating lamina is one of a switchable Bragggrating, a switchable grating recorded in a reverse mode holographicpolymer dispersed liquid crystal, a surface relief grating, and a nonswitching Bragg grating.

In one embodiment the image light has the characteristics of a specklepattern

In one embodiment the eye surface being tracked is at least one of thecornea, lens, iris, sclera and retina.

In one embodiment the detector is a two dimensional array.

In one embodiment the at least one grating lamina encodes at least oneof optical power and diffusing properties.

In one embodiment the detector is connected to an image processingapparatus for determining at least one spatio-temporal characteristic ofan eye movement.

In one embodiment an eye tracker comprises: a waveguide for propagatingillumination light reflected from at least one surface of an eye along awaveguide path; a source of the illumination light; a detector opticallycoupled to the waveguide. The waveguide contains at least one gratinglamina for deflecting illumination light reflected of an eye surfaceinto the second waveguide path towards the detector.

In one embodiment the detector is connected to an image processingapparatus for determining at least one spatio-temporal characteristic ofan eye movement.

In one embodiment the image light is a Purkinje reflection.

In one embodiment the source is a laser.

In one embodiment the source is a light emitting diode.

In one embodiment the illumination grating provides collimated light.

In one embodiment the illumination grating provides divergent light.

In one embodiment the input grating encodes optical power.

In one embodiment the output grating encodes optical power.

In one embodiment at least one of the grating lamina includes at leastone turning grating.

In one embodiment the eye tracker image processing system includes aneural network.

A more complete understanding of the invention can be obtained byconsidering the following detailed description in conjunction with theaccompanying drawings, wherein like index numerals indicate like parts.For purposes of clarity, details relating to technical material that isknown in the technical fields related to the invention have not beendescribed in detail

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic plan view of an eye tracker shown in relation toa human eye in one embodiment of the invention

FIG. 1B is a schematic front elevation view showing elongate gratingelements used in the imaging grating in one embodiment of the invention.

FIG. 1C is a schematic front elevation view showing a two dimensionalarray of grating elements used in the imaging grating in one embodimentof the invention.

FIG. 2 is a schematic plan view of the eye tracker shown the imaging andillumination gratings and input and output gratings in one embodiment ofthe invention.

FIG. 3 is a schematic plan view of the eye tracker shown the imaging andillumination gratings and input and output gratings in one embodiment ofthe invention.

FIG. 4 is a plan view of the eye tracker shown the imaging andillumination gratings and input and output gratings in one embodiment ofthe invention.

FIG. 5 is a schematic cross section view of an illumination grating usedin one embodiment of the invention.

FIG. 6 is a schematic cross section view of an illumination grating usedin one embodiment of the invention.

FIG. 7 is a schematic cross section view of a first aspect of an imaginggrating used in one embodiment of the invention.

FIG. 8 is a schematic cross section view of a second aspect of animaging grating used in one embodiment of the invention.

FIG. 9A is a schematic top elevation view of a first layer of a twolayer imaging grating in one embodiment of the invention.

FIG. 9B is a schematic plan view of a first layer of a two layer imaginggrating in one embodiment of the invention.

FIG. 9C is a schematic side elevation view of a first layer of a twolayer imaging grating in one embodiment of the invention.

FIG. 10A is a schematic top elevation view of a second layer of a twolayer imaging grating in one embodiment of the invention.

FIG. 10B is a schematic plan view of a second layer of a two layerimaging grating in one embodiment of the invention.

FIG. 10C is a schematic side elevation view of a second layer of a twolayer imaging grating in one embodiment of the invention.

FIG. 11A is a schematic cross view of the human eye illustrating theformation of the Purkinje images.

FIG. 11B is a schematic cross view of the human eye illustratingreflections from the retina and iris.

FIG. 12A is a schematic plan view illustrating a first aspect of thelocalization of an eye feature by a two layer imaging grating each layercomprising elongate elements with the elements of the two gratings atright angle.

FIG. 12B is a schematic plan view illustrating a second aspect of thelocalization of an eye feature by a two layer imaging grating each layercomprising elongate elements with the elements of the two gratings atright angle.

FIG. 13A is a schematic cross of the human eye in a first rotationalstate showing a typical speckle pattern formed by the cornea and retina.

FIG. 13B is a schematic cross of the human eye in a first rotationalstate showing a typical speckle pattern formed by the cornea and retina.

FIG. 14 is a schematic front elevation view of a human eye show showingthe directions of motions of speckle patterns produced by the retina andcornea.

FIG. 15A is a schematic cross section view of a human eye in a firstrotational state showing reflection from the retina and cornea.

FIG. 15B is a schematic cross section view of a human eye in a secondrotational state showing reflection from the retina and cornea.

FIG. 16 is a schematic cross section view of an imaging gratingcomprising an array of SBG lens elements with focal length varyingacross the exit pupil in one embodiment of the invention.

FIG. 17A is a schematic cross section view of an imaging gratingcomprising an array of variable power lenses in one embodiment of theinvention.

FIG. 17B is a detail of FIG. 17A showing a variable power lenscomprising a variable index layer and a diffractive element of fixedfocal length.

FIG. 18 is a schematic illustrate of an imaging grating in oneembodiment of the invention in which the imaging grating comprises anarray of interspersed grating elements having at least two differentprescriptions.

FIG. 19 is a schematic cross section view of the illumination grating ofan eye tracker using separate illumination and imaging gratings in oneembodiment of the invention.

FIG. 20 is a schematic plan view the illumination grating of an eyetracker using separate illumination and imaging gratings in oneembodiment of the invention.

FIG. 21 is a schematic cross section view of an alternative illuminationgrating of an eye tracker using separate illumination and imaginggratings in one embodiment of the invention.

FIG. 22A is a schematic plan view of the imaging grating, the imagesampling grating and the detector module of an eye tracker usingseparate illumination and imaging gratings in one embodiment of theinvention.

FIG. 22B is a schematic plan view of image sampling grating and thedetector module of an eye tracker using separate illumination andimaging gratings in one embodiment of the invention.

FIG. 22C is a schematic cross section view of the imaging grating andthe image sampling grating of an eye tracker using separate illuminationand imaging gratings in one embodiment of the invention.

FIG. 22D is a schematic cross section view of image sampling grating andthe detector module of an eye tracker using separate illumination andimaging gratings in one embodiment of the invention.

FIG. 22E is a schematic cross section view of the imaging grating, theimage sampling grating and the detector module of an eye tracker usingseparate illumination and imaging gratings in one embodiment of theinvention.

FIG. 23 is a block diagram showing the principal modules of an eyetracker system using separate illumination and imaging gratings in oneembodiment of the invention.

FIG. 24 is a schematic illustration of a grating element switchingscheme provided by the imaging grating and image sampling grating in oneembodiment of the invention.

FIG. 25 is a schematic plan view of an eye tracker using commonillumination and imaging gratings in one embodiment of the invention.

FIG. 26 is a schematic cross section view showing the imaging andillumination grating and the input, output, image sampling and detectorsampling gratings of an eye tracker using common illumination andimaging gratings in one embodiment of the invention.

FIG. 27A is a schematic plan view of the image sampling grating of aneye tracker using common illumination and imaging gratings in oneembodiment of the invention.

FIG. 27B is a schematic cross section view of the illumination samplinggrating, the input grating and laser of an eye tracker using commonillumination and imaging gratings in one embodiment of the invention.

FIG. 27C is a schematic plan view image sampling grating and thedetector module with detector overlaid of an eye tracker using commonillumination and imaging gratings in one embodiment of the invention.

FIG. 27D is a schematic plan side elevation view showing the imagesampling grating and detector of an eye tracker using commonillumination and imaging gratings in one embodiment of the invention.

FIG. 27E is a schematic cross section view of the output grating and theimage sampling grating an eye tracker using common illumination andimaging gratings in one embodiment of the invention.

FIG. 27F is a schematic cross section view of the input grating and theillumination sampling of an eye tracker using common illumination andimaging gratings in one embodiment of the invention.

FIG. 28 is a simplified representation of the imaging process an eyetracker using common illumination and imaging gratings in one embodimentof the invention.

FIG. 29 provides a system block diagram showing the key modules of aneye tracker using common illumination and imaging gratings in oneembodiment of the invention.

FIG. 30 is a flow chart showing the process for determining eyedisplacement vectors from the recorded speckle data.

FIG. 31A is a flowchart for a calibration process for an eye trackerusing common illumination and imaging gratings in one embodiment of theinvention.

FIG. 31B is a schematic illustration of the initial calibrationprocedure used for an eye tracker in one embodiment of the invention.

FIG. 32 is a schematic plan view of an eye tracker including anillumination sampling grating and an image sampling grating each basedon gratings with grating vectors substantially aligned parallel to thewaveguide plane in one embodiment of the invention.

FIG. 33A is a schematic plan view illustrating a first aspect of an eyetracker including an illumination sampling grating and an image samplinggrating in which the illumination light is angularly offset from theimage light.

FIG. 33B is a schematic plan view illustrating a second aspect of an eyetracker including an illumination sampling grating and an image samplinggrating in which the illumination light is angularly offset from theimage light.

FIG. 34 is a block diagram showing the principal modules of an eyetracker system including a neural network in one embodiment of theinvention.

FIG. 35 is a block diagram showing the principal modules of an eyetracker system based common illumination and imaging grating in whichthe processing system includes a neural network in one embodiment of theinvention.

DETAILED DESCRIPTION

The invention will now be further described by way of example only withreference to the accompanying drawings. It will apparent to thoseskilled in the art that the present invention may be practiced with someor all of the present invention as disclosed in the followingdescription. For the purposes of explaining the invention well-knownfeatures of optical technology known to those skilled in the art ofoptical design and visual displays have been omitted or simplified inorder not to obscure the basic principles of the invention. Unlessotherwise stated the term “on-axis” in relation to a ray or a beamdirection refers to propagation parallel to an axis normal to thesurfaces of the optical components described in relation to theinvention. In the following description the terms light, ray, beam anddirection may be used interchangeably and in association with each otherto indicate the direction of propagation of light energy alongrectilinear trajectories. Parts of the following description will bepresented using terminology commonly employed by those skilled in theart of optical design. It should also be noted that in the followingdescription of the invention repeated usage of the phrase “in oneembodiment” does not necessarily refer to the same embodiment.

The proposed eye tracker aims to satisfy a suite of challengingrequirements. Since it will eventually be integrated into a head-worndisplay, it should make minimum impact on the overall opticalperformance. The inventors' design goals are: a field of view (FOV) of60° horizontal ×48° vertical; 17 mm eye relief; and eye motion box/exitpupil (20 mm.×10-15 mm). Moreover, the eye tracker must satisfy eyesafety requirements for near-eye visual displays with regard to weight(minimal), center of gravity (ergonomic), and profile. Furthermore itshould not compromise: pixel resolution, see-through (≥90%) and powerconsumption (minimal).

Eye Trackers based on classical Purkinje imaging methods suffer fromhigh latency resulting mainly from the large delay incurred by featurerecognition and tracking algorithms. The inventors are stronglymotivated by a desire to develop an eye tracker that firstly simplifiesthe image processing problems of classical eye tracking that oftenresult in unacceptably high latency and secondly can make use ofrelatively unsophisticated detector technology. The proposed eye trackeravoids the cost and complexity of implementing classical Purkinjeimaging methods by tracking eye signatures using low resolution highspeed image sensors. Ideally the detector technology would be equivalentin specification to that used in the infrared mouse a device which isnow ubiquitous and, more importantly, capable of being manufacturedusing sub dollar components. The signatures do not need to be images ofeye features such as pupil edges but can be random structures such asspeckle patterns (including reflections from multiple surfaces andscatter from the optical media inside the eye). However, it is importantthat whatever signature is tracked has a strong spatio-temporalvariation with gaze direction.

The inventors believe that this approach offers significant advantagesin terms of detector resolution, processing overhead and powerconsumption.

An eye tracker according to the principles of the invention provides aninfrared illumination channel for delivering infrared illumination tothe eye and an imaging channel for forming an image of the eye at asensor. In one embodiment of the invention illustrated in FIGS. 1-2 ,the eye tracker comprises a waveguide 100 for propagating illuminationlight towards an eye 116 and propagating image light reflected from atleast one surface of an eye; a light source 112 optically coupled to thewaveguide; and a detector 113 optically coupled to the waveguide.Disposed in the waveguide are: at least one input grating 114 fordeflecting illumination light from the source into a first waveguidepath; at least one illumination grating 102 for deflecting theillumination light towards the eye; at least one imaging grating 101 fordeflecting the image light into a second waveguide path; and at leastone output grating 115 for deflecting the image light towards thedetector. The inventors also refer to the waveguide 100 as the DigiLens.The illumination and imaging gratings are arrays of switchable beamdeflection grating elements with the preferred grating technology beinga SBG as described above. In one embodiment of the invention shown inFIG. 1B the grating elements in the imaging grating 120 are elongate asindicated by 121 with longer dimension orthogonal to the beampropagation direction. In one embodiment of the invention shown in FIG.1C the imaging grating may comprise a two dimensional array 122 ofelements 123 each having optical power in two orthogonal planes.Typically the first and second waveguide paths, that is, the imaging andillumination paths in the waveguide are in opposing directions asillustrated in FIG. 1A. The illumination light will typically be fullycollimated while the image light will have some divergence of angledetermined by the scattering angle from eye services, the angularbandwidth of the gratings and the numerical aperture of the gratingelements. As will be discussed later, in one embodiment the imaging andillumination gratings are provided by a single grating with theillumination and imaging ray paths. Where separate imaging andillumination gratings are used the two gratings may employ different TIRangles within the waveguide. This is advantageous in terms of avoidingthe risk of cross coupling of illumination light into the detector andimage light into the light source.

In FIG. 1A the illumination light path is illustrated by the light 1010from the source which is directed into a TIR path 1011 by the inputgrating and diffracted out of the waveguide as the light generallyindicated by 1012. Typically the eye tracker will have a pupil of size20-30 mm. to allow capture of light reflected from the eye to continueshould the waveguide change position relative to the eye. Since the eyetracker will usually be implemented as part of a HMD its pupil shoulddesirably match that of the HMD. FIG. 1A shows return light 1013reflected from the front surface of the cornea 117 and light 1014reflected from the retina 118. The corneal and retinal image lightenters the waveguide along tray paths such 1015, 1116 and is deflectedinto a TIR path such as 1017 by an active element of the imaging gratingwhich is switched one element at a time. The light 1017 is deflectedinto a ray path 1018 toward the detector by the output grating.Advantageously, the detector reads out the image signal in synchronismwith the switching of the SBG lens array elements. The detector isconnected to an image processing apparatus for determining at least onespatio-temporal characteristic of an eye movement. The image processor,which is not illustrated, detects pre-defined features of thebackscattered signals from the cornea and retina. For example, the imageprocessor may be used to determine the centroid of an eye feature suchas the pupil. Other trackable features of the eye will be well known tothose skilled in arts of eye tracker design and visual optics.

The eye surfaces used for tracking are not necessarily limited to thefront surface of the cornea and the retina. The invention can be appliedusing reflections from any of the surfaces of the lens, iris and scleraincluding any of the reflections normally referred to as Purkinjereflections. In one particularly important embodiment of the inventionto be discussed later the light reflected from the eye is speckle. Thespeckle may arise from reflections at any of the above surfaces or fromthe bulk medium of the cornea lens and other parts of the eye.

Advantageously, the light source is a laser emitting in the infraredband. Typically, the laser emits at a wavelength in the range 785-950nm. The choice of wavelength will depend on laser efficiency, signal tonoise and eye safety considerations. Light Emitting Diodes (LEDs) mayalso be used. In one embodiment of the invention the detector is a twodimensional array. However other types of detector may be used includinglinear arrays and analogue devices such as position sensing detectors.In the embodiment shown in FIG. 1 the illumination grating providesdivergent light. In alternative embodiments of the invention theillumination grating provides collimated light.

The gratings may be implemented as lamina within or adjacent an externalsurface of the waveguide. In other words the grating may be disposedadjacent an optical surface of the waveguide. comprising at least one ofan internal surface or an external surface of the waveguide. For thepurposes of discussing the invention we will consider Bragg gratingsdisposed within the waveguide. Advantageously the gratings areswitchable Bragg gratings (SBGs). In certain embodiments of theinvention passive gratings may be used. However, passive gratings lackthe advantage of being able to direct illumination and collect imagelight from precisely defined areas of the pupil. In one embodiment thegratings are reverse mode SBGs. Although the invention is discussed inrelation to transmission gratings it should be apparent to those skilledin the art that equivalent embodiments using reflection gratings shouldbe feasible in most cases. The gratings may be surface relief gratings.However, such gratings will be inferior to Bragg gratings in terms oftheir optical efficiency and angular/wavelength selectivity.

The input and illumination gratings may be configured in many differentways. FIG. 2 is a schematic plan view showing one possibleimplementation for use with the embodiment of FIG. 1 . Here the inputgrating comprises two grating elements 114A,114B and the illuminationgrating is also divided into the upper and lower gratings 120A,120B,each providing narrow beam deflecting grating strips above and below theimaging grating 102. The detector grating 115 is also indicated. Sincethe guided beams in the input and illumination grating are collimated,and likewise the guided beams in the imaging and detector gratings,there is no cross talk between the two regions of the waveguide.

In the embodiment of the invention shown in FIGS. 3-4 , which is similarto the one of FIG. 2 , the upper and lower illumination grating may bearrays of switchable grating elements 121A,121B comprising switchablegrating elements such as 122A,122B. The SBG deflector arrays scrollillumination across the exit pupil in step with the activation of theimaging grating elements. Finally, in the embodiment of FIG. 4 theillumination grating comprises just one strip 123 containing elements124 disposed along the top edge of the imaging grating.

The invention does not assume any particular configuration of thegrating elements. It is important to note that the SBGs are formed ascontinuous lamina. Hence the illumination gratings elements may beconsidered to be part of the imaging grating. This is a significantadvantage in terms of fabrication and overall form factor. Inembodiments where the illumination grating is split into two elements asdiscussed above the input laser light may be provided by one laser withthe upper and lower beam being provided by a beam splitting means.Alternatively, two separate laser modules may be used to provide lightthat is coupled into the waveguide via the input gratings 114A,114B areillustrated in FIGS. 3-4 . The invention does not assume any particularmethod for providing the laser input illumination or coupling the laserlight into the waveguide. Many alternative schemes should be apparent tothose skilled in the art of optical design.

The illumination grating may provide illumination light of any beamgeometry. For example, the light may be a parallel beam emitted normallyto the surface of the eye tracker waveguide. The illuminator grating isillustrated in more detail in the schematic side elevation view of FIG.5 in which the SBG linear array 130 is sandwiched between transparentsubstrates 130A,130B. Note that the substrate layers extended to coverthe entire waveguide and therefore also act as the substrates for theimaging grating. Advantageously, the ITO layers are applied to theopposing surfaces of the substrates with at least one ITO layer beingpatterned such that SBG elements may be switched selectively. Thesubstrates and SBG array together form a light guide. Each SBG arrayelement has a unique optical prescription designed such that input lightincident in a first direction is diffracted into output lightpropagating in a second direction. FIG. 5 shows TIR illumination beam1020 being deflected by the active element 131 to provide divergentillumination light 1021.

An alternative embodiment of the linear deflector array is shown in theschematic side elevation view of FIG. 6 . In this cases the array 132sandwiched by substrates 132A,132B is based on a lossy grating thatdiffracts incrementally increasing fractions of the guided beam out ofthe waveguide towards the eye. Beam portions 1023A-1023C diffracted bythe grating elements 133A-133C are illustrated. Typically, the indexmodulation of the grating elements will be designed to provide uniformextraction along the array and hence uniform output illumination. Notethat the geometrical optics of FIGS. 5-6 has been simplified for thesake of simplifying the description.

Advantageously, the illumination grating elements may encode opticalpower to provide sufficient beam spread to fill the exit pupil withlight. A similar effect may be produce by encoding diffusioncharacteristics into the gratings. The apparatus may further comprise anarray of passive holographic beam-shaping diffusers applied to thesubstrate, overlapping the linear SBG array, to enhance the diffusion.Methods for encoding beam deflection and diffusion into diffractivedevices are well known to those skilled in the art of diffractiveoptics. Cross talk between the imaging and illumination channels isovercome by configuring the SBGs such that the illumination TIR pathwithin the eye tracker lies outside the imaging TIR path.

In one embodiment of the invention the imaging grating may also encodeoptical power. A two layer SBG imaging grating that encodes opticalpower is illustrated in FIGS. 7-10 . The arrays are shown in theirstacked configuration in FIG. 7 . The substrates 136A,136B and 139A,139Btogether provide the imaging waveguide as illustrated in FIG. 8 wherethe ray path from the eye into the waveguide via an activated SBGelement 42 is represented by rays 1025-1028. The arrays are shown infront, plan and side elevation views in FIGS. 9-10 . The arrays compriselinear arrays of column elements each having the optical characteristicsof a cylindrical lens. The column vectors in the two arrays areorthogonal. The first array comprises the SBG array 135 sandwiched bythe substrates 136A,136B with one particular element 137 beingindicated. The second array comprises the SBG array 40 sandwiched by thesubstrates 139A,139B with one particular element 141 being indicated.

FIG. 11A illustrates the principles of the formation of the first fourPurkinje images corresponding to reflections off the front of the cornea1033,1043; the back of the cornea 1032, 1042; the front of the eye lens1031,1041; and the back of the eye lens 1030,1040. FIG. 11B illustratesthe formation of images of the retina by rays 1034,1044 and the iris byrays 1035,1045.

FIG. 12 shows how the first and second SBG lens arrays of FIGS. 7-10 maybe used to localize an eye feature such as by scanning row and columnSBG elements such as 142 and 143.

With regard to the use of speckle as an eye signature FIG. 13illustrates how the size of speckle feature as recorded in two capturedspeckle images may vary with the eye orientation and displacement withrespect to the eye optical axis 1050. FIG. 13A illustrates speckleformed by illuminating the eye along the direction 1050A which isinitially parallel to the eye optical axis. The components of thecorneal and retinal speckle light parallel to the eye optical axis areindicated by 1050B,1050C. FIG. 14A shows the formation of speckle withthe eye rotated in the plane of the drawing. The detected corneal andretinal speckle light 1050D,1050E parallel to the direction 1050 whichis now no longer parallel to the eye optical axis is shown. As shown bythe insets 1051,1053 the size and spatial distribution of the speckleschanges as the eye rotates. Correlation of the two speckle patterns willprovide a measure of the eye rotation. Note that, typically, the specklepatterns recorded at the detector will combine separate speckle patternsfrom the cornea and retina as well as other surfaces and biologicalmedia interacting with the illumination beam. In one embodiment of theinvention the eye tracker processor compares the speckle images due tolight being scattered from the retina and cornea. When the eye is pannedhorizontally or vertically the relative position of the speckle patternfrom the cornea and retina change accordingly allowing the direction ofgaze to be determined from the relative trajectories of the reflectedlight beams.

FIG. 14 represents the front of the eye 146 cornea 147 and illuminatedregion 148 of the retina illustrates the direction of movement ofcorneal and retinal speckle features as indicated by the vectors 149,150correspond to the ocular displaces illustrated in FIG. 15 . In generalthe ray reflection vectors directions will be closely linked to eyerotation. FIG. 15A represents the reflection of rays from the cornea1056,1057 and retina 1054,1055 for one eye position. FIG. 15B shows thereflection paths from the cornea 1058,1059 and the retina 1060,1061after a horizontal (or vertical) eye rotation. Reflection from thecornea has a strong secular component. Retinal reflection is morediffuse. The size of the corneal reflected angles would ordinarilyrequire a large angular separation between the illumination anddetection optical axes. This would make eye tracking using cornealreflections over large FOVs very difficult. The invention avoids theproblem of imaging large reflection angles (and dealing with are lateraland vertical eye movements which can arise from slippage) by usingmatched scrolling illumination and detection. Hence the reflection anglebecomes relatively small and can be approximated to: Ψ˜2[(D/r−1)Φ+d/r]where r is the cornea radius Φ is the eye rotation and D is the distanceof the eye centre from the displaced centre of curvature of the corneaand d is the lateral displacement of the eye centre.

In one embodiment of the invention based on the one of FIGS. 7-10 theimaging grating comprises an SBG array 143 in which the lens elements144 have varying focal length across the exit pupil. In the embodimentof FIG. 16 grating elements of first and second focal length indicatedby the divergent beams 1062,1064 and 1063,1065 are uniformlyinterspersed. In one embodiment illustrated in FIG. 17A the imagingwaveguide comprises arrays 145 of variable power lens elements 146. Asshown in the detail of FIG. 17B a variable power lens would be providedby combining a diffractive element 147 of fixed focal length with avariable index layer 148.

In one embodiment of the invention shown in the schematic view of FIG.18 the imaging grating comprises a single layer two dimensional SBGarray. A group of elements labelled 152 which comprises interspersedelements such as 153,154. The group forms the image region 151 at thedetector 110. Each SBG element is characterized by one from a set of atleast two different prescriptions. FIG. 18 does not show the details ofthe waveguide and the illumination and input/output gratings. At leastone of the SBG prescriptions corresponds to a lens for forming an imageof the eye on the detector. At least one prescription is optimized forimaging a signature formed by a surface of the eye. Hence the embodimentof FIG. 18 allows eye tracking to be performed using speckle patternsand conventional features such as Purkinje reflections.

FIGS. 19-24 provide schematic illustrations of aspects of an eye trackerbased on the principles of FIGS. 1-6 . In this embodiment of theinvention the earlier described imaging, illumination, input and outputgratings are augmented by an additional grating to be referred to as animage sampling grating which overlays the output grating. FIG. 19 showsa side elevation view of the illumination grating 163. FIG. 20 is a planview showing the imaging grating 165, the illumination grating 163 andthe image sampling grating 170 overlaid on the output grating 164. FIG.21 is a side elevation view of an alternative embodiment of theillumination grating 163. FIG. 22A is a plan view of the imaginggrating, the image sampling grating 14 and the detector module 180. FIG.22B is a plan view of the image sampling grating and the detectormodule. FIG. 22C is a cross sectional view showing the imaging gratingand the image sampling grating. FIG. 22D is a cross sectional view ofthe image sampling grating and the detector module. Finally, FIG. 22E isa cross sectional view of the imaging grating, the image samplinggrating and the detector module. To assist the reader the projectionplane of each illustration is referred to a Cartesian XYZ referenceframe.

The imaging grating 165 comprises an array of column-shaped SBGelements, such as the one labelled 167, sandwiched by substrates168,169. Column elements of the imaging grating 165 are switched on andoff in scrolling fashion backwards and forward along the directionindicated by the block arrow 1320 in FIG. 20 such that only one SBGcolumn is in its diffractive state at any time.

The illuminator array 163 is shown in detail in FIG. 19 comprisessubstrates 161A,161B sandwiching an array of SBG rectangular elementssuch as 163A,163B. The SBG elements may have identical diffractingcharacteristics or, as shown in FIG. 19 , may have characteristics thatvary with position along the array. For example, the element 163Aprovides a diffusion distribution 1310 centered on a vector at ninetydegrees to the array containing rays such as 1311. However, the element63B provides an angled distribution 1312 containing rays such as 1313.In an alternative embodiment shown in FIG. 21 the diffusion polardistributions may have central ray directions that varying in a cyclicfashion across the array as indicated by the rays 1313-1318.

The image sampling grating 170, comprising an array of rectangular SBGbeam deflecting elements 173 such as 176 (shown in its diffracting statein FIG. 22C) sandwiched by substrates 174,175. The waveguide containingthe imaging grating 165, illumination grating 163 and the output grating164 is separated from the image sampling grating 170 by a medium (notillustrated) which may be air or a low refractive index transparentmaterial such as a nanoporous material.

Infrared light from a surface of the eye is coupled into the waveguideby an active imaging grating element, that is, by a diffracting SBGcolumn. The guided beam undergoes TIR in the waveguide up to the outputgrating. As shown in FIG. 22C the output grating 164 deflects the beamthrough ninety degrees into the direction 1322 towards the imagesampling grating 170. As shown in FIG. 22C a portion of the beam 1322 isdeflected into the image sampling grating by an active SBG element 176where it undergoes TIR in the direction indicated by the ray 1323 (andalso by block arrow 1321 in FIG. 20 ). The light that is not sampled bythe image sampling grating indicated by 1320 1321 is trapped by asuitable absorbing material, which is not illustrated. The TIR beam isdeflected in the detector module 180 by a first holographic lens 172 toprovide out image light 1325. Turning now to FIG. 22D we see that thedetector module contains mirror surfaces 177A,177B and a furtherholographic lens 178 which forms an image of the eye features or specklepattern that is being tracked on the detector array 166. Note theholographic lens 172,178 may be replaced by equivalent diffractiveelements based on Bragg or surfaces relief gratings. Conventionalrefractive lens elements may also be used where size constraints permit.

FIG. 23 is a system block diagram of the eye tracker of FIGS. 19-22 .The system modules comprise the imaging grating 300, illuminationgrating 301, illumination grating driver 302, illumination samplinggrating 303, imaging grating driver 304, detector driver 30,image-sampling array driver 306, detector 166 and processor 307. Theapparatus also comprises a laser driver which is not illustrated. Theoptical links from the image grating to the image sampling array and theimage sampling array to the detector are indicated by the block arrows329,330. The processor 307 comprises a frame store 308 or other imagememory device for the storage of captured eye image or speckle patternframes and an image processor 309 further comprising hardware orsoftware modules for noise subtraction 310 and image analysis 311. Theprocessor further comprises hardware control module 312 for controllingthe illumination, imaging and image sampling grating drivers, all saidmodules operating under the control of a main processor 313. Data andcontrol links between components of the system are indicated by 319-325.In particular, each driver module contains switching circuitryschematically indicated by 326-328 for switching the SBG elements in theimaging grating, illumination grating array, and image sampling grating.

FIG. 24 illustrates the switching scheme used in the imaging grating andimage sampling grating. The illumination grating elements are switchedin phase with the imaging grating columns. Column element 165A of theimaging grating array 165 and element 170A of the readout array 170 arein their diffracting states. The projection (indicated by 170B) ofelement 170A on the column 65A defines an active detection aperture.Using such as scheme it is possible to track features of the eye using aX,Y localization algorithm aided by predictions obtained from analysisof displacement vectors determined from successive frames. Methods forimplementing such search schemes will be known to those skilled in theart. The invention does not rely on any particular algorithm orprocessing platform.

FIGS. 25-27 provide schematic illustrations of aspects of an eye trackerthat extends the embodiment of FIGS. 19-24 by introducing a furthergrating component to be referred to as an illumination sampling gratingwhich overlays the input grating. The other feature of this embodimentis that the illumination grating is no longer separate from the imaginggratings. Instead the two are combined in a bi-directional waveguide inwhich a common switchable column grating is used to illuminate and imagethe eye with the illumination and image wave-guided light propagating inopposing directions. The combined gratings will be referred to as theillumination and imaging grating. As will be explained below thefunction of the illumination sampling grating, which is similar instructure to the image sampling grating, is to concentrate the availableillumination into region of the eye selected by the image samplinggrating. This confers the dual benefits of light efficiency andavoidance of stray light from regions of the eye that are not beingtracked.

Turning now to the drawings, FIG. 25 is a plan view showing the imagingand illumination grating 190, the image sampling grating 194,illumination sampling grating 195 the input grating 193 and outputgrating 192 and the detector module 200. Column elements of theillumination and imaging grating are switched on and off in scrollingfashion backwards and forward such that only one SBG column is in itsdiffractive state at any time. The counter propagating beam paths areindicated by 1341,1342. FIG. 26 shows the components of FIG. 25 in aside elevation view. FIG. 27A is a plan view of the illuminationsampling grating. FIG. 27B is a cross sectional view of the illuminationsampling grating 195 including the input grating 193 and the laser 205.FIG. 27C is a plan view of the image sampling grating 194 showing thedetector module 200 and detector 205 overlaid.

FIG. 27D is a side elevation view showing detector module 200 in moredetail. The detector 205 and a cross section of the image samplinggrating 194 are included. FIG. 27E is a cross sectional view of theoutput grating 192 and the image sampling grating 194. FIG. 27F is across section view of the input grating 193 and the illuminationsampling grating 194. To assist the reader the projection plane of eachillustration is referred to a Cartesian XYZ reference frame.

The illumination and imaging grating comprises the array 190 ofcolumn-shaped SBG elements, such as the one labelled 191 sandwiched bythe transparent substrates 190A,190B. The input and output grating whichare disposed in the same layer are labelled by 193,192 respectively. Thedetector module 200 is delineated by a dotted line in FIGS. 25-26 and inmore detail in FIG. 27D. The image sampling grating 194, comprises anarray of rectangular SBG beam deflecting elements (such as 197)sandwiched by substrates 194A,194B. Typically the imaging grating andimage sampling grating are separated by a medium 198 which may be air ora low refractive index transparent material such as a nanoporousmaterial. The illumination sampling grating 195 which is has a verysimilar architecture to the image sampling grating comprises an array ofrectangular SBG beam deflecting elements (such as 196) sandwiched bysubstrates 195A,195B. Typically the imaging grating and image samplinggrating are separated by a medium 199 which may be air or a lowrefractive index transparent material such as a nanoporous material.

Referring to FIG. 26 and FIG. 27F illumination light 1350 from the laseris directed into the illumination sampling grating by a coupling grating207. The light then proceeds along a TIR path as indicated by 1350A,1350B up to an active element 208 where it is diffracted into thedirection 1351 towards the input grating. Not that the image samplinggrating directs all of the illumination light through the active elementof the illumination sampling grating the elements of which are switchedin synchronism with the elements of the image sampling grating to ensurethat at any time the only the region of the that is being imagedreceives illumination. The illumination path in the waveguide isindicated by 1352-1354.

Infrared light 1356 (also illustrated as the signature 1355) from one ormore surfaces of the eye is coupled into the waveguide by a diffractingSBG column such as 191. The guided beam indicated by 1357,1358 undergoesTIR in the waveguide up to the output grating 192. The output gratingdeflects the beam through ninety degree into the direction 1359 towardsthe image sampling grating. As shown in FIG. 27E the beam in direction1359 is deflected into the image sampling grating by an active SBGelement 197 where it undergoes TIR along the ray path indicated by 1360,1361. The TIR beam is deflected into the detector module 200 as light1363 by a first holographic lens 203. Any light that is not sampled bythe image sampling grating is trapped by a suitable absorbing material,which is not illustrated. The absorbing material may be a prism, prismarray, an infrared absorbing coating or some other means known to thoseskilled in the art.

The detector module contains mirror surfaces 201,202 and a furtherholographic lens 204 which forms an image of the eye signature that isbeing tracked on the detector array 205. The ray path from the imagesampling grating to the detector is indicated by the rays 1363-1365.Advantageously, the mirror surfaces are coatings applied to opposingfaces of a prismatic element. However, the invention does not rely onany particular scheme for steering the image light towards the detectorarray. Note that the holographic lens 203,204 may be replaced byequivalent diffractive elements based on Bragg or surfaces reliefgratings. Conventional refractive lens elements may also be used wheresize constraints permit.

In one embodiment of the invention illumination light from laser moduleis converted into S-polarized light which is coupled into the eyetracker waveguide by the input grating. This light is then convertedinto circularly polarized light using a quarter wave plate. An activeSBG column will then diffract the P-component of the circularlypolarized wave guided light towards the eye, the remaining P-polarizedlight being collected in a light trap. The P-polarized light reflectedback from the eye (which will be substantially P-polarized) is thendiffracted into a return TIR path by the active SBG column and proceedsto the detector module as described above. This scheme ensures thatimage and illumination light is not inadvertently coupled into the inputand output gratings respectively. In other embodiments of the inventionthe unwanted coupling of the image and illumination light may beovercome by optimizing the TIR angles, the angular bandwidths of theimaging and illumination gratings, the spacings along the waveguide ofthe input and output gratings, and the illumination and imaging beamcross sections. In one embodiment the illumination light which willtypically in most embodiments of the invention be collimated may beangled such that the waveguide propagation angle of the illuminationbeam differs from the waveguide angles of the image light.

An important feature of the invention is that elements of theillumination sampling grating are switched to allow illumination to belocalized to a small region within the active column of the DigiLensensuring that the illumination is concentrated exactly where it isneeded. This also avoids stray light reflections a problem which canconsume significant image processing resources in conventional eyetracker designs. Since the illumination is scrolled the cornea andretina are not exposed to continuous IR exposure allowing higherexposures levels to be used leading to higher SNR. A safety interlockwhich is not illustrated may be included to switch off the laser when notracking activity has been detected for a predefined time.

The proposed scheme for switching the columns and readout elements inthe embodiments of FIGS. 25-27 is based on tracking the movement of thepupil using a X,Y localization algorithm similar to the one illustratedin FIG. 24 which shows how the ith activated column of DigiLens and jthactivated element of the readout array are used to select the specklepattern region (X,Y).

FIG. 28 is a simplified representation of the detection path startingwith the collimated rays 1400 from an active column element 370 of theimaging array. The rays 1400 are sampled by an element 371 of thedetector grating to provide the rays 1402 which are imaged by theholographic lens 372 to provide the rays 1403 incident on the detector205.

FIG. 29 provides a system block diagram of the eye tracker of FIGS.26-27 . The system modules comprise the illumination and imaging grating190, image sampling grating 194, illumination sampling grating 195,detector 205, laser 206, illumination sampling array driver 340, imagesampling array driver 341, detector driver 342, laser driver 343,illumination and imaging grating driver 344 and processor 345. Theprocessor 345 comprises a frame store or other image storage media 346for the storage of captured eye image or speckle pattern frames and animage processor 347 further comprising hardware or software modules fornoise subtraction 348 and image analysis 349. The processor furthercomprises hardware control module 350 for controlling the illumination,imaging and image sampling grating drivers, all said modules operatingunder the control of a main processor 351. The above described modulesare connected by communication and control links schematically indicatedby 360-369 include control lines for switching the SBG elements in theimaging grating, illumination sampling grating array, and image samplinggrating 367-369.

In one embodiment of the invention the detector array is a detectorarray of resolution 16×16 with a framing rate of 2300 fps of the typecommonly used in infrared mouse equipment. In alternative embodiessimilar sensor technology of resolution 64×64 operating at 670 fps maybe used. The selection of a particular sensor will depend on factorssuch as the required tracking resolution and accuracy and the updaterate of the eye tracker. Exemplary sensors are manufactured by PixartInc. The detector optical prescription will be determined by a detailedray-tracing analysis and will require trade-offs of speckle size,F-number and DigiLens column width. In the case of speckle tracking thedetector lens aperture defines the limiting speckle size. The detectorfield of view is determined by the detector size and the detector lensfocal length. However, the invention could be applied with any currentlyavailable imaging sensor technology. In one embodiment the DigiLensprovides 25 SBG scrolling columns×17 SBG readout elements. The Agilentdevice can be programmed to switch 2300 fps So a complete scan of theFOV will take (25×17)/2300 s.=185 ms. However, in practice the eyetracker will use a more sophisticated X-Y search process that localizesthe pupil using column and readout element coordinates. It isanticipated that on average around 10 search steps may be needed toconverge on the pupil position resulting in a latency of 4.3 ms. On thisbasis the latency of the tracker is potentially ×100 lower than that ofcomparable image processing-based Purkinje-type eye trackers. It is alsoanticipated that the correlation process will be implemented in hardwareresulting in a relatively modest data processing latency. The detectedeye signature is stored and compared with other saved patterns todetermine the eye gaze trajectory and to make absolute determinations ofthe gaze direction (bore sighting). Initial calibration (that is,building up the database of saved patterns) is carried out by directingthe user to look at test targets at predefined points in the field ofview (FOV) over which the eye gaze is to be tracked. Since the framesare of low resolution large numbers of samples may be collected withoutsignificant computational overhead.

Although the invention may be used to detect any type of eye signature,speckle is attractive because it avoids the image analysis problems ofidentifying and tracking recognizable features of the eye that areencountered in Purkinje imaging schemes. Prerequisites for measuring eyedisplacement vectors (rotational and/or translational) include achievingan adequate level of speckle contrast (after detector noise and ambientlight have been subtracted from the detected signal) and being able toresolve individual speckle grains. A high signal to noise ratio (SNR) isessential for detecting variations in speckle properties at requiredangular resolution. The SNR depends on the speckle contrast, which isdefined as the ratio of the root means square (rms) variation of thespeckle intensity to the mean intensity. The speckle contrast liesbetween 0-1 assuming Gaussian statistics. The detector should have lownoise and a short integration time. If the motion of the eye isappreciably faster than the exposure time of the CCD camera rapidintensity fluctuations of the speckle pattern will occur, the average ofthe detected patterns resulting in a blurred image with reduced specklecontrast. The smallest speckle size is set by the diffraction limit.Applying the well-known formula from diffraction theory: w=˜2.44D/a(assuming: a detector lens to detector distance D˜70 mm.; IR wavelength1=785 nm.; and detector lens aperture a˜3 mm) we obtain a diffractionlimited speckle diameter w at the detector of ˜64 microns. Theresolution of a typical mouse sensor is around 400-800 counts per inch(cpi), with rates of motion up to 14 inches per second (fps). Hence thelimiting speckle size is equivalent to one count per 64 micron at 400cpiwhich is roughly compatible with the expected speckle size.

The proposed strategy for processing speckle data captured by the eyetracker is based on the following assumptions.

Speckle patterns provide unique “fingerprints” of regions of the corneaand retina.

Unlike speckle interferometry which requires that the speckle motion isless than speckle size, speckle imaging using a detector array requiresthat the speckle displacement from frame to frame is greater than thespeckle size

A displacement of the cornea and retina relative to the detector willresult in a shift of the speckle pattern by the same amount

The shifts of the corneal and retinal speckle patterns will be inopposite directions.

The motion of the speckles can be determined from the correlation of twoconsecutive frame speckle patterns. This information together with therelative motion of the corneal and retinal speckle patterns can be usedto determine eye displacement vectors.

The correlation and image analysis processes may take advantage standardtechniques already developed in applications such as radar, biologicalimaging etc.

The speckle contrast and speckle size at the detector are compatiblewith the detector resolution and SNR.

The following characteristics of the speckle image may also be used toassist the tracking of the eye use speckle: speckle grain size; specklebrightness (either individual or collective brightness); speckle shape;rate of change of any of the preceding characteristics with ocularmovement; and relative directions of corneal and retinal bemadisplacements. It is further recognized that each of these aspects ofthe speckle image will be dependent on the illumination beam direction(scanning or static); the detection optics and the focal length of theimaging optics. The rate of change of the corneal versus retinalspeckles will depend on the focal length.

The flow chart in FIG. 30 summarizes the process for determining eyedisplacement vectors from the recorded speckle data. The process relieson a database of frame data collected during initial calibration andnoise characteristics. The calculation of the displacement vectors usesinputs from a suite of mathematical models that simulate the first ordereye optics, the eye tracker optics and the eye dynamics. The process maybe interrupted by the user or automatically when a DigiLens failureoccurs. The process also includes DigiLens hardware control to enableX,Y addressing of DigiLens columns and readout elements. The correlationprocess for obtaining the eye displacement vector from two detectedframes in one embodiment may be summarized as follows. Each frame issubdivided into small sub frames. The sub-frame coordinates may bepredefined or alternatively may be determined by an interactive schemeusing the output from an Eye Dynamics Model. A 2D correlation mapbetween the sub images from the two frames is calculated starting with aone pixel step in the x and y directions and repeat the calculationincreasing the step size by one pixel at a time. Other statisticalmetrics may also be computed at this stage to assist in refining thecalculation. We then repeat the correlation process for another selectedframe region. A displacement vector is then computed using (for the timeperiod between the two analyzed frames) using the peaks of thecorrelation maps. Ideally the sub frames should be entirely within thecorneal or retinal fields, the two being distinguished by their opposingdirections. Data which does not yield clear separation of the two willbe rejected) at this stage. The calculation is refined using data froman Eye Optical Model which models of the eye dynamics and an Eye TrackerModel which models the optical system. The verified displacement vectoris used to determine the next search X,Y coordinates (ie SBG column,row) for the Eye Tracker using predicted gaze trajectory calculatedusing an Eye Dynamics Model. The basic ray optics used in the Eye Modelin particular the relationship of the first order corneal and retinalreflection paths of the eye may be modelled using ray-tracing programssuch as ZEMAX. Standard eye models well known to those skilled in theart will be adequate for this purpose. Further models may be used tosimulate speckle from the retina and the cornea. The Eye Dynamics Modelcarries out a statistical analysis of the displacement vectors fromprevious frames to determine the most optical next X,Y search location(ie the columns and readout elements to be activated in the DigiLens.

Initial calibration is carried out by directing the user to look at testtargets at predefined points in the FOV. The bore-sighting process isillustrated in FIG. 32 which shows a flowchart (FIG. 32A) and aschematic illustrates of the initial calibration procedure (FIG. 32B).According to FIG. 31A the bore sighting procedure 400 comprises thefollowing steps:

At step 401 present targets to the eye at location j;

At step 402 capture a series of frames at location j;

At step 403 store the capture frames;

At step 404 move to the next target position in the field of view (FOV);

At step 405 repeat the process while j is less than a predefined integerN; otherwise end the process (at step 406).

Referring to FIG. 31B we see that initial calibration will be carried bypresenting targets (typically lights sources, resolution targets etc.)to the viewer at different points 1≤j≤N in the field of view 410 (thepoint also being labelled as 411-413) and capturing and storing framesof signature images at each location. The targets may be presentedsequentially along the sweep path labelled by 414. However, otherpresentation schemes may be used. The stored frames will be processed toenhance SNR and extract statistical metrics (such as histograms,probability density functions for speckle size etc.) for subsequent“on-the-fly” frame comparison. Each frame provides a “fingerprint” forthe region of the FOV concerned. The signatures will vary in: relativepositions of the corneal and retinal reflections, or where specklepatterns are used: speckle contrast; and speckle size distribution(which is linked to optical magnification).

In relation to the embodiment of FIG. 25 we have described the use of animage sampling grating overlaying the output grating. The image samplinggrating comprises a linear array of switchable grating elements, eachelement when in its diffracting state sampling a portion of the light inthe waveguide and deflecting it along the image sampling grating towardssaid detector. In a similar fashion an illumination sampling gratingoverlays the input grating. The illumination sampling grating isoptically coupled to the light source and comprises a linear array ofswitchable grating elements. Each element when in its diffracting statedeflects light from the illumination sampling grating into thewaveguide. Turning to FIG. 32 we next consider an embodiment thatimplements image and illuminations sampling grating using a singlegrating layer. The eye tracker 420 comprises a DigiLens 420, imagesampling gating 422 illumination sampling grating 423 containingelements such as 424 and 425 respectively. Output and input gratings426,427 link the sampling gratings to the detector and light sourcesrespectively. As indicated by the shading pattern of the gratingelements each element comprising a switchable grating with Bragg fringesslanted at 45 degrees with grating vectors in the plane of the drawing;that is, in a plane parallel to the waveguiding surfaces. The inventorsrefer to these gratings as turning gratings. Hence illumination ray 1422undergoing TIR in the DigiLens is deflected through an angle of ninetydegrees by the active element 425 into the ray direction 1423. Similarlythe image ray 1420 is deflected through an angle of ninety degrees inthe direction 1421 by the active element 424. It should also be apparentfrom consideration of the drawing that all of the gratings may be formedin a single layer in a single waveguide (with the appropriate electrodepatterning of the sandwiching substrates. It should also be apparentthat the turning grating principle may be applied in any of the abovedescribed embodiments including those in which the DigiLens comprisesseparated overlapping illumination and imaging gratings. The samplinggratings may overlap. The design of the turning gratings may be based onthe teachings of U.S. Pat. No. 8,233,204 entitled OPTICAL DISPLAYS whichis incorporated herein by reference in its entirety.

A challenge in a single layer eye tracker design of the type describedabove is to provide adequate eye illumination without compromising theability of the DigiLens to collected scattered light from the eye. Mostattempts to use gratings for light management in bi-directionalwaveguides fail because of the fundamental principle of gratingreciprocity. In practical terms this means that some of the image lightalmost always ends up getting coupled into the illumination path to thesource by the input grating. In the reciprocal process some of theillumination light is diffracted into the imaging path to the detectorby the output grating. The amount of this cross coupling will depend onthe beam divergence and waveguide dimensions. The proposed solutionwhich is illustrated in FIG. 33 assumes the common illumination andimaging waveguide architecture discussed above and, in particular, theone illustrated in FIG. 25 . The apparatus comprises the DigiLens 450which comprises an array of SBG columns such as 451 and a waveguidecomponent 451 comprising the illumination sampling and imaging samplinggratings 452,453 and containing grating elements (which we may refer toas pixels) such as 454,455. A cross section of the illumination samplinggrating is provided by 456. The cross section of the DigiLens is alsoshown and is indicated by 458. Gratings for coupling the image andillumination light to the detector and laser are indicated by 458,459.Finally, an eye is represented by 460. The input and output gratings,which will typically overlap the sampling gratings as discussed earlier,are not illustrated. We next consider the ray paths, first defining anormal to the illumination waveguide as indicated by 1430. The path ofan incident beam at an angle U1 up the eye is indicated by the rays1431-1436 comprising the TIR path 1432, coupling into the DigiLens viathe active element 455 as indicated by the ray 1433, propagating up tothe active column element 451 as indicated by ray 1434, diffractiontowards the eye along 1435, and light 1436 striking a surface of theeye. The reflection light path from the eye to the detector is indicatedby the rays 1437-1440 with scattered light from the eye indicated by1437 entering the DigiLens as 1438 and propagating along the path 1439before being diffracted into the image sampling grating via the element454 and proceeding along the path 1440 leading the detector. FIG. 33Bshows the corresponding ray paths 1441,1442 for an incident ray 1441launched at the angle U2 (greater than U1) which terminates at thedetector, the ray paths following the logic of FIG. 33A. In oneembodiment of the invention the method illustrated in FIG. 33 eliminatesunwanted light coupling by applying a small tilt to the input beam angleby an amount equivalent to at least 1 pixel of the eye tracker imagingmatrix, for a specular beam. In other embodiments larger pixel offsetsmay be useful for better discrimination. A similar tilt is required inthe case of diffuse beams. Gratings are currently the preferred optionfor producing the tilt. However, alternative methods based on prisms maybe used. In one embodiment the method illustrated in FIG. 33 is used toprovide different grating tilts for the upper and lower halves of thewaveguide, thereby preventing over sizing of the lower portion of thewaveguide.

In the description of the eye tracker data processing architecture wehave discussed how initial calibration will be carried by presentingtargets (typically lights sources, resolution targets etc.) to theviewer at different points in the field of view and capturing andstoring frames of speckle pattern images at each location. These imagesare used aid the processing of live data when the eye tracker is normaluse. It is proposed that the process could be aided by incorporating anartificial neural network within the processor. The bore sightingprocess would correspond to training the networks. The network could beused to compensate at least part of any systematic measurements errorsoccurring in the processing. In one embodiment of the invention shown inthe block diagram of FIG. 34 the eye tracker system comprises the eyetracker waveguide 430, detector 431, processor comprising: mainprocessor 432, waveguide SBG control module 433, neural network 434 andimage database 435. The system modules are connected by communicationand control links referenced by numerals 436-442. A more detailedarchitecture incorporating a neural network is shown in FIG. 35 This isarchitecture is intended for use with a common illumination and imaginggrating eye tracker designs such as the one of FIG. 25 .

Although the description of the invention has emphasized the detectionof speckle patterns it should be apparent from consideration of thedescription and drawings that the same optical architecture and indeedmany features of the processing architecture may be used to perform eyetracking using other optical signatures from the eye. For examplefeatures such as bright or dark pupils and glint may provide suitablesignatures. The blurring of the eye feature being tracked does notpresent an impediment providing that the detected image contains enoughcontent for correlations to be made between captured frames and storedimages capture in the bore sighting (or neural network training) stage.

The optical design requires careful balancing of the high source fluxneeded to overcome throughput inefficiencies arising from the smallcollection angles, low transmission thorough the DigiLens and the lowreflectivity of the eye (˜2.5% at the surface of the cornea) with therequirement for eye-safe IR illumination levels. Typically, forapplications in which the eye tracker is used for hours at a time undercontinuous IR exposure the eye irradiance should not exceed around 1mW/cm2. The appropriate standards for eye safe infrared irradiance arewell known to those skilled in the art. Since the proposed eye trackerscrolls the illumination across the eye the cornea and retina are notexposed to continuous IR exposure allowing higher exposures levels to beused leading to higher speckle contrast level and therefore higher SNRat the detector. In a switchable grating based design there is the riskof a switching malfunction causing the laser beam scanning to freezeresulting in all of the available output laser power being concentratedinto a small area of the eye. To overcome this problem a safetyinterlock may be provided to switch off the laser when no trackingactivity has been detected for a predefined time of, typically, a fewminutes. During this dead time the IR exposure may be allowed toincrease significantly without exceeding the safety threshold.

The proposed eye tracker avoids the cost and complexity of implementingclassical Purkinje imaging methods by tracking eye signatures using lowresolution high speed image sensors. The signatures do not need to beimages of eye features such as pupil edges but can be random structuressuch as speckle patterns. However, it is important that whateversignature is tracked has a strong spatio-temporal variation with gazedirection. Conventional iris image capture systems are an indicator thelevel of processing that will be required in an eye tracker. The irisimage is typically acquired by a camera using infrared light in the 700nm-900 nm band resolving in the region of 100-200 pixels along the irisdiameter. The first step is usually to detect and remove stray lightbefore proceeding to determine the boundaries of the iris. Typically thecenters and radii of iris and pupil are approximated initially byapplying a circular edge detector. High accuracy and rapid responsetimes require high-performance and high-cost microprocessors that arebeyond the scope of consumer products. Traditional image processingdesigns based on software are too slow. It is known that significantimprovements may result from an iris recognition algorithms based on ahardware-software co-design using low-cost FPGAs. The systemarchitecture consists of a 32-bit general purpose microprocessor andseveral dedicated hardware units. The microprocessor executes insoftware the less computationally intensive tasks, whereas thecoprocessors speed-up the functions that have higher computational cost.Typically, depending on the function implemented, coprocessors speed-upthe processing time by a factor greater than 10 compared to its softwareexecution. However, the best latency achieved with hardware-softwareco-designs, is typically in the range 500-1000 ms. It should be notedthat an eye tracker is a much more demanding proposition for an imageprocessor. Detecting a clean iris image is only the first step. Applyingthe edge detection algorithms as the eye moves around the eye box willrequire several frames to be analyzed adding to the overall latency.

The proposed eye tracker is compatible with many display applications inconsumer products, avionics and other fields such as Augmented Realityby enabling the features of: wide field of view; large exit pupil; thinform factor; low inertia; and easy integration with near-eye displaytechnologies.

It should be emphasized that the drawings are exemplary and that thedimensions have been exaggerated. For example thicknesses of the SBGlayers have been greatly exaggerated.

In any of the above embodiments the waveguides may be curved or formedfrom a mosaic of planar or curved facets.

An eye tracker based on any of the above-described embodiments may beimplemented using plastic substrates using the materials and processesdisclosed in PCT Application No.: PCT/GB2012/000680, entitledIMPROVEMENTS TO HOLOGRAPHIC POLYMER DISPERSED LIQUID CRYSTAL MATERIALSAND DEVICES.

Advantageously, the SBGs are recorded in a reverse mode HPDLC materialin which the diffracting state of SBG occurs when an electric field isapplied across the electrodes. An eye tracker based on any of theabove-described embodiments may be implemented using reverse modematerials and processes disclosed in PCT Application No.:PCT/GB2012/000680, entitled IMPROVEMENTS TO HOLOGRAPHIC POLYMERDISPERSED LIQUID CRYSTAL MATERIALS AND DEVICES.

While the invention may be applied with gratings of any type includingswitching or non-switching gratings based on Bragg (volume) holograms,or surface-relief gratings the preferred grating technology is a SBG,which offers the advantages of fast switching, high optical efficiencyand transparency and high index modulation.

With regard to the use of grating arrays it should be appreciated thenumber of elements used in an array need not be very large, depending onthe FOV over which gaze is to be tracked.

It should also be noted that the gratings used in the eye tracker arenot necessarily all switching gratings. Switching gratings may be usedin combination with passive grating technologies. As has been indicatedby the description and drawings more than one grating layer (lamina) maybe used. The grating layers discussed above are SBGs disposed betweeninternal waveguide surfaces (or in other words sandwiched betweentransparent substrates that combine to form the waveguide. However inequivalent embodiments some of the gratings layers could be applied toexternal waveguide surfaces. This would apply in the case of surfacerelief gratings.

A glass waveguide in air will propagate light by total internalreflection if the internal incidence angle is greater than about 42degrees. Thus the invention may be implemented using transmissiongratings if the internal incidence angles are in the range of 42 toabout 70 degrees, in which case the light extracted from the light guideby the gratings will be predominantly p-polarized.

Using sufficiently thin substrates the eye tracker could be implementedas a long clear strip appliqué running from the nasal to ear ends of aHMD with a small illumination module continuing laser dies, light guidesand display drive chip tucked into the sidewall of the eyeglass. Astandard index matched glue would be used to fix the display to thesurfaces of the HMD.

The method of fabricating the SBG pixel elements and the ITO electrodesused in any of the above-described embodiments of the invention may bebased on the process disclosed in the PCT Application No. US2006/043938,entitled METHOD AND APPARATUS FOR PROVIDING A TRANSPARENT DISPLAY.

The invention does not rely on any particular methods for introducinglight from a laser source into the eye tracker and directing lightscattered from the eye onto a detector. In the preferred embodiments ofthe invention gratings are used to perform the above functions. Thegratings may be non switchable gratings. The gratings may be holographicoptical elements. The gratings may be switchable gratings.Alternatively, prismatic elements may be used. The invention does notrely on any particular method for coupling light into the display.

It should be understood by those skilled in the art that while thepresent invention has been described with reference to exemplaryembodiments, it is to be understood that the invention is not limited tothe disclosed exemplary embodiments. Various modifications,combinations, sub-combinations and alterations may occur depending ondesign requirements and other factors insofar as they are within thescope of the appended claims or the equivalents thereof.

What is claimed is:
 1. An apparatus for tracking the movement of an eye,comprising: a first waveguide; a light source for illuminating said eye;at least one detector optically coupled to said waveguide; disposedwithin said waveguide, at least one grating layer comprising at leastone grating elements with a grating prescription for diffracting lightreflected from said eye into a total internal reflection (TIR) path tosaid detector; and a processor configured to receive signals output fromsaid detector, wherein said detector is configured to record a firstoptical signature resulting from a first gaze direction and to record asecond optical signature resulting from a second gaze direction, andwherein said processor is configured to determine changes in gazedirection by correlating said first optical signature and said secondoptical signature.
 2. The apparatus of claim 1, wherein each of saidfirst optical signature and said second optical signature results fromat least one of reflection or scatter from at least one of an opticalsurface or an ocular medium of said eye.
 3. The apparatus of claim 1,wherein each of said first optical signature and said second opticalsignature is characterized by at least one of signature feature spatialsize or signature feature spatial distribution.
 4. The apparatus ofclaim 3, wherein each of said first optical signature and said secondoptical signature is further characterized by at least one of speckle orglint.
 5. The apparatus of claim 1, wherein said first waveguidecontains a grating or prism for coupling illumination from said sourceinto an illumination TIR path in said first waveguide and a grating forcoupling light out of said illumination TIR path onto said eye.
 6. Theapparatus of claim 1, further comprising a second waveguide supporting agrating or prism for coupling illumination from said source into anillumination TIR path in said second waveguide and a grating forcoupling light out of said illumination TIR path onto said eye.
 7. Theapparatus of claim 1, wherein at least one of said first plurality ofgrating elements or said second plurality of grating elements comprisesat least one switchable grating element having a diffracting state and anon-diffracting state.
 8. The apparatus of claim 1, wherein at least oneof said first plurality of grating elements or said second plurality ofgrating elements comprises at least one switchable grating elementhaving a diffracting state and a non-diffracting state, wherein saidelement in said diffracting state deflects light reflected from said eyeinto at least one of said first or second TIR paths.
 9. The apparatus ofclaim 1, wherein at least one of said first plurality of gratingelements or said second plurality of grating elements comprises an arrayof elongate elements.
 10. The apparatus of claim 1, wherein at least oneof said first plurality of grating elements or said second plurality ofgrating elements is a two-dimensional array.
 11. The apparatus of claim1, wherein saki grating layer comprises at least one of a switchableBragg grating, a switchable grating recorded in a reverse modeholographic polymer dispersed liquid crystal, a switchable gratingrecorded in a reverse mode holographic polymer dispersed liquid crystal,a surface relief grating or a non-switching Bragg grating.
 12. Theapparatus of claim 1, wherein said eye surface is at least one of acornea, front or rear lens surface, iris, sclera or retina.
 13. Theapparatus of claim 1, wherein said detector is a two-dimensional array.14. The apparatus of claim 1, further comprising at least one gratingwith at least one of optical power or diffusing properties.
 15. Theapparatus of claim 1, wherein said detector is connected to at least oneof an image processing apparatus for determining at least onespatio-temporal characteristic of an eye movement or an image processingsystem including a neural network.
 16. The apparatus of claim 1, whereinsaid detector is coupled to said waveguide by a grating or a prism. 17.The apparatus of claim 1, integrated within a HMD or HUD.
 18. Theapparatus of claim 1, further comprising a turning grating.
 19. Theapparatus of claim 1, wherein said source is a laser or a light emittingdiode.
 20. The apparatus of claim 1, wherein the processor is configuredto determine changes in the gaze direction using the relativedisplacements of a detected optical signature resulting from light beingscattered from the retina and a detected optical signature resultingfrom light being scattered from the cornea.